Skip to content

Query-level resource usages tracking#13172

Merged
jed326 merged 7 commits intoopensearch-project:mainfrom
ansjcy:query-level-resource-usages-tracking
Jun 7, 2024
Merged

Query-level resource usages tracking#13172
jed326 merged 7 commits intoopensearch-project:mainfrom
ansjcy:query-level-resource-usages-tracking

Conversation

@ansjcy
Copy link
Copy Markdown
Member

@ansjcy ansjcy commented Apr 12, 2024 •
edited
Loading

Description

  • Instrument resource usages before a task finishes on a data node, more specifically, get resource usages before a phase response is sent from query/fetch/.. phase.
  • Piggyback the resource usages data with shard search response headers.
  • Gather data node search tasks resource usages from headers, on coordinator node, and store them into the search context, so that we can infer the query-level resource usage.

Related Issues

Resolves #12399

benchmark tests

Did extensive benchmark tests, merged the tests results by calculating average on multiple runs, and here are the test results:

baseline-resulsts.txt
feature-ressults.txt

I don't see significant impact on search latency with this change.

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)
  • Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions github-actions bot added enhancement Enhancement or improvement to existing feature or request Search:Query Insights labels Apr 12, 2024
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 12, 2024

❌ Gradle check result for 32f9a75: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 12, 2024

Compatibility status:

Checks if related components are compatible with change 32f9a75

Incompatible components

Incompatible components: [https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/sql.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/common-utils.git]

@ansjcy ansjcy changed the title Query-level resource usages tracking [DRAFT] Query-level resource usages tracking Apr 15, 2024
@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Jun 22, 2024
Open
@BrewTestBot BrewTestBot mentioned this pull request Jun 26, 2024
Merged
This was referenced Jun 19, 2024
Closed
Closed
Open
Open
Closed
Open
@Bukhtawar Bukhtawar mentioned this pull request Aug 19, 2024
Merged
This was referenced Sep 6, 2024
Closed
Open
Closed
Open
Open
Open
This was referenced Sep 28, 2024
Closed
Open
@ansjcy ansjcy mentioned this pull request Oct 31, 2024
Merged
3 tasks
@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Nov 6, 2024
Open
@andrross andrross mentioned this pull request Dec 2, 2024
Open
3 tasks
@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Dec 16, 2024
Closed
@ansjcy ansjcy mentioned this pull request Feb 20, 2025
Open
@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Apr 14, 2025
Open
@ansjcy ansjcy mentioned this pull request Apr 15, 2025
Open
@opensearch-ci-bot opensearch-ci-bot mentioned this pull request May 8, 2025
Closed
@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Jul 31, 2025
Closed
@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Nov 17, 2025
Open
This was referenced Jan 13, 2026
Merged
Closed
@manojfaria manojfaria mentioned this pull request Feb 20, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rishabhmaurya rishabhmaurya rishabhmaurya left review comments

@msfroh msfroh msfroh approved these changes

@jainankitk jainankitk jainankitk approved these changes

@getsaurabh02 getsaurabh02 getsaurabh02 approved these changes

@abbashus abbashus Awaiting requested review from abbashus

@adnapibar adnapibar Awaiting requested review from adnapibar

@anasalkouz anasalkouz Awaiting requested review from anasalkouz anasalkouz is a code owner

@andrross andrross Awaiting requested review from andrross andrross is a code owner

@Bukhtawar Bukhtawar Awaiting requested review from Bukhtawar Bukhtawar is a code owner

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE is a code owner

@dblock dblock Awaiting requested review from dblock

@dbwiddis dbwiddis Awaiting requested review from dbwiddis dbwiddis is a code owner

@dreamer-89 dreamer-89 Awaiting requested review from dreamer-89

@gbbafna gbbafna Awaiting requested review from gbbafna gbbafna is a code owner

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal is a code owner

@mch2 mch2 Awaiting requested review from mch2 mch2 is a code owner

@nknize nknize Awaiting requested review from nknize

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 is a code owner

@reta reta Awaiting requested review from reta reta is a code owner

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 is a code owner

@ryanbogan ryanbogan Awaiting requested review from ryanbogan

@sachinpkale sachinpkale Awaiting requested review from sachinpkale sachinpkale is a code owner

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli is a code owner

@shwetathareja shwetathareja Awaiting requested review from shwetathareja shwetathareja is a code owner

@sohami sohami Awaiting requested review from sohami sohami is a code owner

@tlfeng tlfeng Awaiting requested review from tlfeng

@VachaShah VachaShah Awaiting requested review from VachaShah

+2 more reviewers

@kaushalmahi12 kaushalmahi12 kaushalmahi12 left review comments

@deshsidd deshsidd deshsidd approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

backport 2.x Backport to 2.x branch backport-failed enhancement Enhancement or improvement to existing feature or request Search:Query Insights v2.15.0 Issues and PRs related to version 2.15.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Query Insights] Capture query-level resource usage metrics

9 participants

@ansjcy @rramachand21 @jainankitk @msfroh @rishabhmaurya @deshsidd @getsaurabh02 @kaushalmahi12 @jed326